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Program description 


Talent Search aims to help low-income and first-generation 
college students (those whose parents do not have four-year 
college degrees) complete high school and gain access to 
college through a combination of services designed to improve 


academic achievement and increase access to financial aid. Ser- 
vices include test taking and study skills assistance, academic 
advising, tutoring, career development, college campus visits, 
and financial aid application assistance. 



Research Two studies of Talent Search met What Works Clearinghouse about 5,000 Talent Search participants, as well as a comparison 
(WWC) evidence standards with reservations — one conducted sample of more than 70,000 students created through propen- 

in Texas and another in Florida. Together, the studies included sity score matching.^ 



Effectiveness Talent Search was found to have potentially positive effects on completing school. 

Staying in schooi Progressing in schooi Compieting schooi 

Rating of effectiveness N/A N/A Potentially positive effects 

Improvement index^ N/A N/A Average: +17 percentile points 

Range: +14 to +19 percentile points 



1. The evidence presented in this report is based on avaiiabie research. Findings and conciusions may change as new research becomes avaiiable. 

2. These numbers show the average and range of improvement indices for aii findings across the two studies. 
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The Talent Search study summarized in this intervention report 
was prepared by staff of Mathematica Policy Research, Inc. 
(MPR). Because the principal investigator for the WWC dropout 
prevention review is also an MPR staff member, the study was 



rated by staff members from Caliber Associates, who also 
prepared the intervention report. The report was then reviewed 
by MPR staff members and by members of the WWC Technical 
Review Team and external peer reviewers. 



Developer and contact 

Talent Search is one of several federal programs established by 
and funded through the Higher Education Act of 1965. These 
programs — known as “TRIO programs” because there were 
originally three of them — are designed to promote college 
enrollment and completion among low-income students. More 
information about Talent Search can be found at http://www. 
ed.gov/Droqrams/triotalent/index.html . 

Scope of use 

Currently, Talent Search serves about 380,000 students a year 
through more than 400 projects sponsored by institutions of 
higher education, public and private agencies or organizations, 
and some secondary schools. 

Description of intervention 

Talent Search aims to promote high school graduation and col- 
lege enrollment through academic support, exposure to college 
campuses, and assistance with understanding and accessing 



financial aid. The program serves both middle and high school 
students. Talent Search projects must primarily serve students 
who are low income and whose parents did not complete a 
four-year college degree. Program services include career 
exploration and aptitude assessment, tutorial services, informa- 
tion on postsecondary education, exposure to college cam- 
puses, counseling, academic advising, information on financial 
aid, help with completing college admissions and financial 
aid applications, assistance in preparing for college entrance 
exams, mentoring programs, and workshops for participants’ 
families. According to a 2000 survey of project directors, nearly 
half of Talent Search participants received 10 or fewer hours of 
services a year. 

Cost 

In fiscal year 2005, Talent Search received approximately $145 
million in federal funding to serve 384,588 participants, for an 
average cost of $376 per participant. In some cases, local pro- 
grams supplement this federal funding with other resources.® 



The WWC reviewed seven studies of the effectiveness of Tal- 
ent Search. Three of these studies were included within one 
research report (Constantine, Seftor, Martin, Silva, & Myers, 
2006). Among the three studies in this report, those conducted 
in Texas and Florida met WWC evidence standards with reserva- 
tions. The third study included in the Constantine et al. (2006) 
study — conducted with data from Indiana — did not assess 
outcomes from any of the three relevant domains for dropout 
prevention (staying in school, progressing in school, and com- 



pleting school). The remaining four Talent Search studies did not 
meet WWC evidence screens. 

The Texas study involved 10 Talent Search projects (each 
serving 10-20 high schools) and 4,027 participants, who 
were matched to 30,842 nonparticipants from the same 
high schools based on propensity scoring methods that 
matched students on 18 demographic, socioeconomic, and 
academic characteristics. The Florida study involved five 
Talent Search projects (each serving 10-20 high schools) and 



3. For more detailed budgetary information on Talent Search since its inception in 1965, see U.S. Department of Education. (2002). A profile of federal TRIO 
programs: 2002. Washington, DC: U.S. Department of Education, Office of Postsecondary Education. 
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Research (continued) 



Effectiveness 



The WWC found 
Talent Search to have 
potentially positive effects 
on completing school 
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900 participants, who were matched to 42,514 nonparticipants 
from the same high schoois using propensity scoring methods 
that matched students on 13 demographic, socioeconomic, 
and academic characteristics."* For both the Texas and Fiorida 
samples, statistical tests found that treatment and comparison 



group sampies were not statistically different at the 0.05 ievel 
on any of the demographic or academic measures used in the 
matching procedures. In both states, the study focused on 
participants who were ninth graders in the fail of the 1995-96 
school year. 



Findings 

The WWC review of interventions for dropout prevention 
addresses student outcomes in three domains: staying in 
school, progressing in schooi, and completing school. 

Completing school. The Texas and Florida studies examined 
the program’s effects on the likelihood that students received 
a high school diploma or General Educational Development 
(GED) certificate within five years of entering ninth grade. 

The Texas study indicated that Talent Search participants 
completed school at a significantly higher rate than comparison 
group students — 86% compared with 77%. The Florida study 
indicated that Talent Search participants completed school at a 
significantly higher rate than comparison group students — 84% 



compared with 70%. Neither study examined the separate effect 
of Talent Search on receipt of high school diplomas compared 
with attainment of GED certificates. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention condition 
and the comparison condition, and the consistency in findings 
across studies (see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations^ The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is entirely based on the size of the effect, regardless of 



the statistical significance of the effect, the study design, or the 
analysis. The improvement index can take on values between 
-50 and +50, with positive numbers denoting favorable results. 
The average improvement index for completing school is +17 
percentile points across the two studies, with a range of +14 to 
+19 percentile points across the findings. 

Summary 

The WWC reviewed seven studies of the effectiveness of Talent 
Search. Two of these studies met WWC standards with reserva- 
tions; the others did not meet evidence or relevance standards. 



4. Information on the number of high schools per project was obtained by the WWC from the study authors. In both the Florida and Texas analyses, weights 
were used to account for the closeness of the match (with closer matches receiving a larger weight) and to account for the fact that students in the Talent 
Search group could be matched to more than one comparison student. 

5. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Talent Search, no corrections for clustering or 
multiple comparisons were needed. 
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The WWC found 
Talent Search to have 
potentially positive 
effects on completing 
school (continued) 

References 



Based on the results from the two qualifying studies, the WWC 
found potentially positive effects of Talent Search on complet- 
ing school. Neither of these studies assessed the program’s 



effectiveness on staying in school or progressing in school. The 
evidence presented In this report is limited and may change as 
new research emerges. 
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Franklin, P. (1985). Flelping disadvantaged youth and adults enter 
college: An assessment of two Federal programs. Washing- 
ton, DC: College Entrance Examination Board.^ 

Green, C. M. (2003). An assessment of the influence of a Talent 
Search program on the academic and career goals of pro- 
gram participants. Dissertation Abstracts International, Wayne 
State University. (UMI No. 3086430).^ 

Simelton, V. E. (1994). An evaluation of the Educational Talent 
Search Program at the University of Arkansas at Little Rock. 
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For more information about specific studies and WWC caicuiations, piease see the WWC Talent Search Technicai 
Appendices . 



6. The study does not use a strong causal design: the study used a nonequivalent comparison group. 

7. The outcome measures examined are not relevant to this review. 
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Appendix 



Appendix A1.1 


Study characteristics: Constantine, Setter, Martin, Siiva, & Myers (2006) — ^Texas study (quasi-experimentai design) 


Characteristic 


Description 


Study citation 


Constantine, J. M., Seftor, N. S., Martin, E. S., Silva, T., & Myers, D. (2006). A study of the effeot of the Talent Search program on secondary and postsecondary outcomes in 
Florida, Indiana, and Texas: Final report from phase II of fhe national evaluation. Report prepared by Mathematica Policy Research for the U.S. Department of Education, Office 
of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Washington, DC: U.S. Department of Educafion. 


Participants 


The Texas study used a quasi-experimentai research design. The sample included 4,027 students in the intervention group and 30,842 students in the comparison group. 
Propensity score modeling was used to match Talent Search students to comparison students who attended the same high schools and who were in the ninth grade in the 
1995-96 school year. Matching was based on 18 demographic and academic characteristics, including students’ eighth-grade test scores, race or ethnicity, socioeconomic 
status, English proficiency, special education status, and enrollment in vocational education programs. There were no statistically significant differences between the 
intervention group and the matched comparison group on the baseline characteristics used in the matching procedures. A Talent Search student could be matched to multiple 
comparison students. Weights were used to account for the closeness of fhe match — with closer matches receiving larger weights.^ In addition, the comparison sample was 
weighted to equal the size of the treatment group so as not to overstate statistical significance. So, fhe infervention and comparison groups each had an effective sample size 
of about 4,000. 

Compared with all Texas high school students. Talent Search participants were more likely to be female (62% compared with 47%), economically disadvantaged (51 % 
compared with 38%), and blacker Hispanic (73% compared with 53%). Tatent Searc/r students were less likely than other high school students in the state to be behind 
grade level (16% compared with 29%), to be receiving special education services (5% compared with 12%), or to score in the bottom quartile on standardized tests (22% 
compared with 27% for math and 20% compared with 27% for reading). 


Setting 


The Texas study was conducted in 10 Talent Search projects (each including 10-20 high schools) and included participants who entered ninth grade in 1 995-96. ^ 


Intervention 


Most participants received services in their junior and senior years of high school. Participants were either recruited to participate or volunteered to be in the program. 


Comparison 


Comparison group students did not participate in Talent Search anti attended the same high schools as students in the intervention group. 


Primary outcomes 
and measurement 


One relevant outcome from the Texas study — high school completion rates — is included in this summary. This measure represents whether students earned a high school 
diploma or received a GED certificate. (See Appendix A2 for a more detailed description of this outcome measure.) The study also examined the program's effects on financial 
aid receipt and college enrollment. However, these outcomes do not fall wifhin the three domains (staying in school, progressing in school, and completing school) examined 
by the WWC’s review of dropout prevention interventions. Therefore, these additional outcomes are not included in this report. 


Teacher training 


No specific informafion concerning staff training was provided. 



1 . Additional information on how weights were constructed was obtained by the WWC from the study authors. 

2. Information on the number of high schools per project was obtained by the WWC from the study authors. 
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Appendix A1.2 Study characteristics: Constantine, Seftor, Martin, Siiva, & Myers (2006) — Fiorida study (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Constantine, J. M., Seftor, N. S., Martin, E. S., Silva, T., & Myers, D. (2006). A study of the effeot of the Talent Search program on secondary and postsecondary outcomes in 
Florida, Indiana, and Texas: Final report from phase II of the national evaluation. Report prepared by Mathematica Policy Research for the U.S. Department of Education, Office 
of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Washington, DC: U.S. Department of Education. 


Participants 


The Florida study used a quasi-experimentai research design. The matched sample included 900 students in the intervention group and 42,514 students in the comparison 
group. Propensity score modeling was used to match Talent Search participants to similar students who attended the same high schools and who were in the ninth grade in 
1995-96. Matching was based on 13 demographic and academic characteristics, including whether students were economically disadvantaged, learning disabled, overage 
for grade, emotionally or physically disabled, or enrolled in a gifted or talented program. Students were also matched on gender, race or ethnicity, primary language spoken 
at home, and citizenship status. Intervention and comparison students were not statistically different from each other at the 0.05 level on any measures used in the matching 
procedures. Elowever, there were statistically significant differences at the 0.10 level on two measures — language spoken at home and whether participated in a dropout 
prevention program. These differences were controlled for in regression models used to estimate program impacts. Weights were used to account for the closeness of the 
match, with closer matches receiving larger weights. In addition, the comparison sample was weighted to equal the size of the treatment group so as not to overstate statisti- 
cal significance. So, the intervention and comparison groups each had an effective sample size of 900. 

Compared with all Florida high school students, participants in Talent Search were more likely to be black (46% compared with 25%) and economically disadvantaged (63% 
compared with 37%). They were less likely to be learning disabled (4% compared with 8%), overage for grade (10% compared with 26%), or male (34% compared with 
53%). They were also less likely to be Elispanic (5% compared with 16%) or to speak a language other than English at home (3% compared with 14%). 


Setting 


The Florida study was conducted in five Talent Search projects throughout the state, each including 10-20 high schools.'' Participants included students who entered ninth 
grade in 1995-96. 


Intervention 


Most participants received services in their junior and senior years of high school. School and Talent Search staff recruited participants to participate. 


Comparison 


Comparison group students did not participate in Talent Search anti attended the same high schools as students in the intervention group. 


Primary outcomes 
and measurement 


One relevant outcome from the Florida study — high school completion — is included in this summary. This measure represents whether sample members earned a high 
school diploma or received a GED certificate. (See Appendix A2 for a more detailed description of this outcome measure.) The study also examined the program’s effects on 
financial aid receipt and college enrollment. However, these outcomes do not fall within the three domains (staying in school, progressing in school, and completing school) 
examined by the WWC’s review of dropout prevention interventions. Therefore, these results are not included in this report. 


Teacher training 


No specific information concerning staff training was provided. 



1 . Information on the number of high schools per project was obtained by the WWC from the study authors. 
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Appendix A2 Outcome measures in the compieting schooi domain 



Characteristic 


Description 


Earned a high school 
diploma or GED 


The percentage of students who reoeived their diploma or GED by the end of their fifth academic year after entering the ninth grade. In both Texas and Florida, this information 
comes from a statewide database that traoked public high school enrollment and completion and GED receipt. The measure includes only diplomas earned from public high 
schools within the state and GED certificates earned within the state. Diplomas trom private high schools and credentials earned outside the state are not included. 
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Appendix A3 Summary of study findings inciuded in the rating for the compieting schooi domain^ 









Author’s findings from the study 














Mean outcome 




WWC calculations 












Mean difference^ 


Statistical 






Study 




Talent Search Comparison 


(Talent Search - 


significance^ 


Improvement 


Outcome measure 


sample 


Sample size^ 


group group 


comparison) 


Effect size'* (at a = 0.05) 


index^ 



Constantine et al., 2006 — ^Texas study (quasi-experimental design)^ 



Earned a high school High school 8,054 

diploma or GED students 


86% 


77% 


9% 


0.37 


Statistically 

significant 


+14 


Average^ for completing school (Constantine et al., 2006 — ^Texas study) 








0.37 


Statistically 

significant 


+14 



Constantine et al., 2006 — Florida study (quasi-experimental design)^ 



Earned a high school High school 1,800 84% 

diploma or GED students 


70% 


14% 


0.49 


Statistically 

significant 


+19 


Average^ for completing school (Constantine et al., 2006 — Florida study) 






0.49 


Statistically 

significant 


+19 


Domain average^ for completing school across all studies 






0.43 


na 


+17 



na = not applicable 

1 . This appendix reports findings considered for the effectiveness rating and the improvement index. 

2. This column represents the effective sample size after weighting. Talent Search students could be matched to multiple comparison students. Weights were used in the analysis that made the effective sample size for the comparison 
group equal to the Talent Search group. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. Effect sizes were calculated using the Cox index for dichotomous outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Constantine et al. (2006), no corrections 
for clustering or multiple comparisons were needed. 

8. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. 
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Appendix A4 Talent Search rating for the compieting schooi domain 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negativeJ 
For the outcome domain of completing school, the WWC rated Talent Search as having potentially positive effects. It did not meet the criteria for positive effects 
because both Talent Search studies involved quasi-experimental designs. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and 
negative effects) were not considered because Talent Search was assigned the highest applicable rating. 

Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. The WWC analysis found two studies of Talent Search that demonstrated statistically significant positive effects in the completing school 
domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No studies of Talent Search found statistically significant or substantively important negative effects in the completing school domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies of Talent Search met WWC evidence standards for a strong design. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in the completing school domain. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings 
of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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